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Abstract 


This  report  summarizes  a  workshop  on  analysis  and  evaluation  of  enterprise  architectures  that  was 
held  at  the  Carnegie  Mellon  Software  Engineering  Institute  (SEI)  in  April  of  2010.  The  SEI  in¬ 
vited  accomplished  practitioners  from  government  and  industry  to  discuss  key  issues  in  analyzing 
and  evaluating  enterprise  architectures.  After  several  opening  talks  by  individuals  who  presented 
the  state  of  the  practice  of  enterprise  architecture  within  their  own  organizations,  the  group  consi¬ 
dered  a  series  of  questions,  including  (1)  Is  there  a  fundamental  difference  between  analyzing  and 
evaluating  enterprise  architectures  and  system  of  system  architectures?  (2)  How  are  quality 
attribute  concerns  expressed  and  analyzed  in  practice  for  an  enterprise  architecture?  (3)  How  are 
enterprise  architectures  evaluated  in  practice?  For  each  question,  discussion  included  considera¬ 
tion  of  the  current  state  of  the  practice,  identification  of  difficulties  sufficient  to  motivate  an  or¬ 
ganization  to  seek  a  solution  or  an  alternative  (“pain  points”),  challenges  in  current  practice,  and 
differences  between  government  and  industry  contexts.  This  report  summarizes  the  workshop  di¬ 
alogue  and  findings,  and  presents  a  proposal  for  an  Enterprise  Architecture  Analysis  and  Evalua¬ 
tion  process. 
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1  A  Workshop  on  Analysis  and  Evaluation  of  Enterprise 
Architectures 


1.1  Background 

The  Architecture-Centric  Engineering  team  at  the  Carnegie  Mellon5’  Software  Engineering  Insti¬ 
tute  (SEI)  has  been  extending  its  research  from  software  architectures  into  the  realms  of  software- 
reliant  system  architectures  and  system-of-systems  architectures.  This  work  has  focused  on  ex¬ 
tending  the  principles  of  the  SEI  Quality  Attribute  Workshop  and  the  SEI  Architecture  Tradeoff 
Analysis  Method®  (ATAM®)  to  develop  methods  applicable  to  larger  scale  architectures,  as  re¬ 
ported  by  Gagliardi  and  colleagues  (Gagliardi  2009). 

These  methods  have  been  successfully  applied  to  development  and  acquisition  in  the  DoD  tactical 
systems  domain.  In  2009,  the  Architecture-Centric  Engineering  team  launched  an  effort  to  deter¬ 
mine  whether  the  approach  described  by  Gagliardi  and  colleagues  could  be  applied  to  help  organ¬ 
izations  systematically  analyze  and  evaluate  enterprise  architectures  (Gagliardi  2009). 

This  workshop  was  held  to  help  the  SEI  better  understand  these  issues:  the  state  of  the  practice  in 
enterprise  architecture  (EA),  the  differences  in  the  practice  between  government  and  industry  con¬ 
texts,  and  where  analysis  and  evaluation  methods  fit  into  EA  practice. 

1.2  Workshop  Participants 

This  workshop  assembled  a  group  of  experts  and  practitioners  in  the  field  of  EA.  The  participants 
are  listed  (in  alphabetical  order)  in  Table  1 .  The  participants  were  invited  based  on  their  expertise 
and  interest  in  exploring  (1)  the  challenges  of  defining  the  appropriate  analysis  and  evaluation  me¬ 
thods  for  enterprise  architectures;  (2)  the  timing  of  these  activities  within  the  architecture  develop¬ 
ment  life  cycle;  and  (3)  EA  evaluation  criteria. 


Table  1:  Workshop  Participants 


Name 

Organization 

David  Cuyler 

Sandia  National  Laboratories 

Michael  Gagliardi 

SEI  Architecture  Centric  Engineering  Initiative 

Linda  Parker  Gates 

SEI  Acquisition  Support  Program 

John  Grasso 

Federal  Railroad  Administration 

COL  Michael  Gray 

U.S.  Army  CIO/G-6 

John  Klein 

SEI  Architecture  Centric  Engineering  Initiative 

Ian  Komorowski 

Whitney,  Bradley,  &  Brown,  Inc. 

Donna  Marcum 

Veterans  Health  Administration 

Plamen  Petrov 

Blue  Cross  Blue  Shield  Association 

Todd  Tieger 

Deloitte  &  Touche  LLP 

Jeff  Tyree 

Capital  One 

Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 

Architecture  Tradeoff  Analysis  Method  and  ATAM  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 
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1 .3  About  This  Workshop 


Table  2  shows  the  agenda  for  the  workshop.  After  a  short  welcome  and  overview  that  established 
the  purpose  and  expected  outcomes  of  the  workshop,  several  participants  delivered  short  presenta¬ 
tions  that  described  the  state  of  EA  in  their  own  organizations.  The  SEI  staff  gave  a  short  presen¬ 
tation  on  the  SEI  System  of  Systems  specification  and  evaluation  techniques.  The  material  pre¬ 
sented  by  the  SEI  was  referred  to  frequently  in  the  workshop  discussions,  and  so  is  contained  in 
Appendix  B  of  this  report. 


Table  2:  Workshop  Agenda 


Day  1 

20  April  2010 

Time 

Topic 

Presenter 

0800-0830 

Continental  Breakfast 

0830-0845 

Welcome  and  background;  Workshop  goals,  purpose,  and  ap¬ 
proach 

John  Klein,  SEI 

0900-1130 

Short  invited  presentations  on  the  practice  of  enterprise  architec¬ 
ture,  relevant  to  the  workshop 

Attendees  invited  to 
make  a  presentation 

1130-1200 

Discussion  and  formation  of  working  groups 

All 

1200-1300 

LUNCH 

1300-1700 

Working  groups  meet 

All 

Day  2 

21  April  2010 

Time 

Topic 

Presenter 

0800-0830 

Continental  Breakfast 

0830-0900 

Plenary:  Discussion;  Summary  of  progress  so  far; 

Working  groups  “course  correction”  if  necessary 

All 

0900-1100 

Working  groups  meet 

All 

1100-1200 

Working  groups  prepare  for  presentation 

All 

1200-1330 

Working  Lunch  -  Working  group  presentations  and  discussion 

1330-1400 

Wrap-up  and  next  steps 

All 

On  Day  1 ,  the  workshop  participants  elected  not  to  split  into  two  working  groups,  but  to  operate 
as  a  single  working  group  for  the  entire  workshop.  Therefore  the  preparation  of  the  presentation 
and  delivery  of  the  presentation  on  Day  2  were  essentially  a  single  activity. 

The  single  working  group  addressed  a  series  of  questions,  listed  in  Table  3.  For  each  question, 
there  was  a  discussion  of  the  current  state  of  the  practice  in  this  area.  Participants  then  identified 
deficiencies  and  “pain  points”  in  the  current  practice,  discussed  approaches  and  challenges  to 
changing  current  practice,  and  discussed  whether  and  how  the  issues  differed  in  government  and 
industry  contexts. 


2  CMU/SEI-2010-TN-023 


Table  3:  Workshop  Questions 


Is  there  a  difference  between  the  artifacts  for  an  enterprise  architecture  and  a  system-of-systems  (SoS)  or  system 
architecture? 

How  are  enterprise  architectures  analyzed? 

How  does  an  organization  develop  evaluation  criteria  for  an  enterprise  architecture? 

How  and  when  are  enterprise  architectures  evaluated? 

What  is  “in  scope”  for  analyzing  and  evaluating  enterprise  architectures? 

Can  an  enterprise  architecture  be  viewed  as  a  set  of  system  of  systems?  If  so,  can  existing  SoS  analysis  and  evalua¬ 
tion  methods  be  applied  to  the  enterprise  architecture? 

What  role  do  quality  attributes  play  in  analysis  and  evaluation  of  enterprise  architectures? 


1.4  Organization  of  This  Report 

This  document  summarizes  the  discussions  from  the  workshop.  The  report  is  laid  out  as  follows: 

•  Sections  2  through  4  summarize  the  discussions  of  the  working  group.  As  the  group  dis¬ 
cussed  questions,  recurring  themes  and  crosscutting  issues  emerged,  and  so  this  content  is 
organized  thematically,  rather  than  question  by  question. 

•  Section  5  summarizes  the  findings  of  the  workshop. 

•  Section  6  outlines  how  the  SEI  System  of  Systems  (SoS)  Architecture  Engagement  can 
be  applied  to  enterprise  architectures. 

•  Appendix  A  surveys  definitions  of  EA. 
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2  Definitions  and  Boundaries 


This  section  summarizes  discussions  about  the  definition  of  EA,  boundaries  of  EA,  and  relation¬ 
ship  of  EA  to  the  enterprise  business. 

2.1  Defining  Enterprise  Architecture 

Participants  preferred  to  use  the  definition  of  architecture  from  the  Institute  of  Electrical  and  Elec¬ 
tronic  Engineers  (IEEE)  and  substitute  system  with  enterprise  to  create  a  working  definition  of 
EA  (IEEE  2000). 

•  IEEE- 1471  Definition  of  Architecture:  “The  fundamental  organization  of  a  system  embo¬ 
died  in  its  components,  their  relationships  to  each  other,  and  to  the  environment,  and  the 
principles  guiding  its  design  and  evolution.” 

•  Proposed  Definition  of  Enterprise  Architecture:  “The  fundamental  organization  of  an  en¬ 
terprise  embodied  in  its  components,  their  relationships  to  each  other,  and  to  the  envi¬ 
ronment,  and  the  principles  guiding  its  design  and  evolution.” 

EA  is  different  from  other  architecture  disciplines  (software,  system,  and  system-of-systems)  in 
that  the  enterprise  generally  exists  before  any  EA  activity  is  started,  and  must  continue  to  exist 
and  function  as  it  is  being  changed.  The  evolution  perspective  included  in  the  IEEE  definition 
emphasizes  that  evolution  is  an  essential  consideration  in  EA  in  practice. 

Appendix  A  presents  a  brief  survey  of  other  definitions  of  EA. 

2.2  Relationship  Between  Enterprise  Architecture  and  the  Enterprise  Business 

Many  of  the  participants  supported  the  assertions  that  “enterprise  architecture  is  a  strategic  activi¬ 
ty.”  One  participant  asserted  that  this  strategic  perspective  may  distinguish  EA  from  SoS  and  sys¬ 
tem  architectures. 

EA  plays  a  critical  role  in  supporting  and  informing  the  strategic  decisions  made  within  the  organ¬ 
ization;  however,  participants  report  that  EA  activities  are  typically  underfunded.  This  seems  to 
stem  from  short-term  management  focus,  difficulty  collecting  return-on-investment  data,  and  dif¬ 
ficulty  quantifying  the  value  of  any  strategic  activity. 

One  of  the  participants  showed  Figure  1,  taken  from  the  book  by  Ross,  Weill,  and  Robertson 
(Ross,  Weill,  &  Robertson,  2006). 
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Strategic 

Initiatives 


Strategic 

Initiatives 


Strategic 

Initiatives 


Strategic 

Initiatives 


Defines  strategic  limits 


Learning  & 
exploitation 


Establishes 

priorities 


Foundation  for  execution 

■  Core  Business  Processes 

■  IT  Infrastructure 


Figure  1:  Enterprise  Architecture  Ties  the  Enterprise's  Operating  Model  to  the  Foundation  for 
Execution  Through  an  Engagement  Model  (Ross,  Weill,  &  Robertson,  2006) 


One  participant  proposed  three  operating  modes  for  enterprise  architects  within  the  enterprise: 

1 .  At  the  lowest  level,  enterprise  architects  operate  in  an  urgent  response  mode,  reacting  to 
crises  as  they  arise. 

2.  Next,  enterprise  architects  may  operate  in  a  continuous  improvement  mode,  making  incre¬ 
mental  changes  and  generally  avoiding  crises. 

3.  Finally,  enterprise  architects  may  operate  in  a  transformative  change  mode,  collaborating 
with  business  leaders  to  enable  new  business  capabilities  and  new  business  models. 

In  practice,  enterprise  architects  are  nearly  always  operating  in  all  three  modes,  but  the  most  ef¬ 
fective  organizations  will  spend  less  effort  in  the  urgent  response  and  more  in  transformative 

change. 

2.3  Bounding  Enterprise  Architecture  in  Practice 

The  Open  Group  Architecture  Framework  (TOGAF)  defines  EA  as  comprising  four  domains  (The 

Open  Group  2009): 

1 .  The  business  architecture  defines  the  business  strategy,  governance,  organization,  and  key 
business  processes. 

2.  The  data  architecture  describes  the  structure  of  an  organization’s  logical  and  physical  data 
assets  and  data  management  resources. 
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3.  The  application  architecture  provides  a  blueprint  for  the  individual  application  systems  to 
be  deployed,  their  interactions,  and  their  relationships  to  the  core  business  processes  of  the 
organization. 

4.  The  technology  architecture  describes  the  logical  software  and  hardware  capabilities  that 
are  required  to  support  the  deployment  of  business,  data,  and  application  services.  This  in¬ 
cludes  IT  infrastructure,  middleware,  networks,  communications,  processing,  standards,  and 
so  on. 

Most  participants  considered  “enterprise  architecture”  to  be  typically  implemented  and  practiced 
as  “enterprise  infonnation  systems/information  technology  (IS/IT)  architecture,”  comprising  only 
the  last  three  domains.  An  exception  among  organizations  represented  at  this  workshop  was  the 
Veterans  Health  Administration,  which  is  creating  an  enterprise  business  architecture  that  is  “IT- 
agnostic.” 

The  partition  between  business  architecture  and  the  other  three  domains  seems  to  be  driven  by 
lines  of  authority  and  control  of  the  business  architecture  domain.  An  EA  practice  is  generally 
hosted  by  the  Chief  Information  Officer  or  some  other  IS/IT-oriented  segment  of  the  enterprise. 
Organizational  relationships  and  roles  between  EA-IS/IT  and  business  architecture  within  an  EA 
vary  with  the  organization’s  history,  culture,  and  maturity.  Business  process  modeling  is  typically 
managed  and  performed  outside  of  the  EA  unit,  and  may  be  performed  with  less  precision  than  is 
required  to  support  enterprise  IS/IT  architecture.  In  these  cases,  enterprise  architects  may  do  “just 
enough”  business  modeling  to  support  their  own  needs.  An  organization  that  is  just  starting  to 
document  an  EA  can  recover  the  business  architecture  from  existing  or  legacy  systems  (by  reverse 
engineering  the  business  rules  codified  in  software  and  then  applying  expert  interpretation). 

This  delineation  of  EA  scope  is  a  more  general  issue — various  organizational  structures  and  lines 
of  authority  result  in  the  organization’s  adding  responsibilities  to  or  removing  responsibilities 
from  the  scope  of  the  EA  team. 

One  participant  asserted  that  “EA  is  a  management  discipline,”  and  the  scope  should  include  ac¬ 
tivities  such  as  outsourcing  and  managing  the  supply  chain.  In  the  government  context,  scope 
would  be  extended  to  include  acquisition-related  considerations. 

For  this  workshop,  participants  decided  to  bound  EA  to  “enterprise  information  systems 
/information  technology  (IS/IT)  architecture,”  comprising  the  data,  application,  and  technology 
domains.  Business  architecture  provides  context  and  linkage  from  the  IS/IT  architecture  to  the 
organization’s  business  goals. 
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3  Enterprise  Architecture  Design  and  Documentation 
Practices 


This  section  summarizes  discussions  addressing  the  design  life  cycle  and  artifacts,  to  help  answer 
the  following  questions: 

•  Where  would  architecture  analysis  and  evaluation  fit  into  the  EA  life  cycle? 

•  What  artifacts  would  be  available  to  support  analysis  and  evaluation? 

3.1  Typical  Enterprise  Architecture  Artifacts 

A  roadmap  is  a  plan  that  defines  a  sequence  of  architecture  states  or  transition  architectures  that 
will  change  the  “as-is”  EA  to  a  desired  target  architecture.  Participants  agreed  that  this  roadmap  is 
the  critical  EA  artifact  that  relates  technology  to  business  goals,  and  ties  together  the  many  con¬ 
current  projects  underway  in  an  organization  at  any  time.  A  typical  roadmap  in  industrial  practice 
covers  three  years — it  is  felt  that  projecting  beyond  that  time  frame  is  not  efficient. 

Other  artifacts  typically  developed  and  maintained  included 

•  reference  models 

•  strategic  plans  (for  DoD  enterprise  architectures,  would  include  campaign  plans  and 
posture  statements) 

•  architecture  principles 

•  conceptual  architectures  and  other  architecture  descriptions,  describing  baseline,  target 
and  transition  architectures 

•  architecture  patterns 

In  general,  these  artifacts  are  a  mixture  of  different  types  of  products  (the  DoDAF,  TOGAF,  sys¬ 
tem  architecture,  spreadsheets,  Visio  diagrams,  and  other  tools)  and  reside  in  different  reposito¬ 
ries.  For  the  U.S.  Army,  the  Capability  Architecture  Development  and  Integration  Enviromnent 
(CADIE)  is  the  primary  repository  for  EA  artifacts  (products);  however,  other  information  rele¬ 
vant  to  the  EA  resides  in  other  repositories.  Federation  of  data  from  multiple  repositories  is  neces¬ 
sary  for  making  meaningful  decisions  using  the  EA. 

Given  that  the  participants  stressed  that  the  primary  role  of  EA  is  to  support  enterprise  decision 
making  (refer  to  Section  2.2  above),  artifacts  need  to  be  selected,  developed,  and  maintained  with 
this  role  in  mind. 

During  this  discussion,  it  became  clear  that  the  terminology  used  in  the  DoD,  other  government 
agencies,  and  commercial  practice  are  very  different  and  impact  the  ability  to  apply  principles 
across  domains.  Consistency,  or  at  least  a  comprehensive  mapping,  would  be  a  great  help  to  prac¬ 
titioners. 
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3.2  Enterprise  Architecture  Artifacts  -  Depth  and  Detail 

All  participants  agreed  that  it  is  easy  to  get  bogged  down  in  EA  documentation.  Available  tools 
can  probe  the  data  network  and  crawl  through  the  existing  systems  to  extract  partial  representa¬ 
tions  of  the  as-is  architecture.  It’s  easy  to  add  more  detail  to  these  repositories  manually,  leading 
to  bloated  architecture  documentation  that  is  expensive  and  time  consuming  to  maintain  and  so 
often  becomes  stale.  It  was  noted  that  each  additional  level  of  decomposition  adds  twice  the  effort 
needed  to  complete  the  previous  level.  EA  documentation  carried  down  to  the  level  of  a  technical 
architecture  is  too  detailed  to  be  useful  or  effective.  The  appropriate  level  of  documentation 
comes  back  to  the  use  of  EA  to  support  strategic  decision  making  within  the  organization — the 
documentation  detail  needs  to  be  “just  enough”  to  support  the  decisions  that  the  organization  must 
make. 

Development  tooling  is  necessary  in  any  case — tools  in  use  ranged  from  iServer  (Visio  backed  by 
a  database  repository)  (Orbus  Software  20 1 0)  for  a  small  enterprise,  up  to  more  sophisticated 
tools  such  as  ARIS  (IDS  Scheer  2010),  Casewise  (Casewise  2010),  IBM  System  Architect  (IBM 
2010),  and  Troux  (Troux  Technologies  2010).  Even  the  more  sophisticated  tools  like  Troux  did 
not  readily  support  documenting  tradeoffs  between  design  alternatives.  We  will  discuss  why  do¬ 
cumenting  such  tradeoffs  is  important  in  Section  6. 

A  critical  activity  in  successful  EA  teams  is  communication  about  the  EA  with  the  rest  of  the  en¬ 
terprise.  Several  participants  noted  that  the  representations  produced  by  the  tools  (e.g.,  various 
Unified  Modeling  Language  [UML])  diagrams  and  entity-relationship  diagrams)  are  not  useful  for 
communication  outside  of  the  architecture  team.  One  participant  talked  about  keeping  the  tool 
repository  “in  his  back  pocket”  to  be  used  by  the  EA  team  for  decision  making  but  never  shown  to 
stakeholders.  Alternative  representations — that  are  more  “executive-friendly” — are  needed  for 
communication  with  stakeholders. 

Several  participants  pointed  out  that  the  practice  of  keeping  the  EA  models  up  to  date  with  the 
solution/project  architectures  and  as-built  infrastructure  varies  among  organizations.  Up-to-date 
models  help  new  stakeholders,  but  most  decision  makers  seem  to  quickly  internalize  any  confor¬ 
mance  discrepancies.  Within  an  organization,  some  models  are  kept  current  while  others  are  al¬ 
lowed  to  lag.  In  fact,  workshop  participants  who  had  invested  heavily  in  using  these  tools  to  de¬ 
velop  the  initial  capture  of  the  EA  expressed  the  sentiment  that  while  they  felt  this  work  was 
crucial  to  their  efforts,  ongoing  maintenance  of  the  models  was  not  cost  effective.  There  was 
agreement  that  documentation  of  the  invariants  and  principles  should  be  kept  current. 

3.3  Projects  Are  the  Units  of  Delivery 

The  EA  roadmap  is  realized  by  a  number  of  projects,  each  producing  a  project  or  solution  archi¬ 
tecture.  These  projects  vary  widely  in  scope  and  complexity,  ranging  from  simple  (for  example,  a 
new  version  of  a  vendor  platform  or  package),  to  complex  (for  example,  moving  to  a  new  vendor 
platform  or  adding  a  new  application).  Complexity  should  not  be  confused  with  architecture  signi¬ 
ficance — some  “simple”  projects  are  architecturally  significant,  while  some  “complex”  projects 
may  have  little  or  no  architectural  impact.  The  distinction  is  not  always  explicit — having  a  deci¬ 
sion  model  to  systematically  distinguish  between  the  two  types  of  projects  based  on  architecture 
significance  would  be  useful. 
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A  project  may  also  be  categorized  according  to  the  operating  mode  that  drives  it  (see  Section  2.2 
above).  Projects  may  include  both  re-engineering  of  existing  components  and  packages,  and  de¬ 
velopment  of  new  components  and  packages. 

3.4  Decisions  Are  First-Class  Artifacts 

The  participants  favored  defining  EA  in  terms  of  structures  determined  by  components  and  rela¬ 
tionships.  However,  in  considering  how  to  document  an  EA,  there  was  agreement  that  document¬ 
ing  decisions  as  first-class  artifacts  is  more  important  than  documenting  structure. 

The  process  of  making,  evaluating,  documenting,  and  managing  the  version/configuration  of  deci¬ 
sions  is  at  the  heart  of  the  EA  governance  process. 

One  of  the  participants  presented  the  metamodel  that  his  organization  uses  for  documenting  EA, 
which  includes  decisions  and  rationale  as  explicit  elements  in  the  metamodel. 

3.5  Use  of  Patterns  Guides  Projects 

One  important  type  of  EA  decision  is  the  selection  of  “patterns.”  Projects  are  frequently  specified 
in  terms  of  the  selected  patterns. 

In  practice,  participants’  use  of  the  term  patterns  was  an  extension  of  the  typical  definition  of  ar¬ 
chitecture  patterns.  A  pattern  was  defined  as  a  reference  to  an  already  instantiated  set  of  architec¬ 
ture  elements  and  relationships  in  their  enterprise — an  exemplar  of  how  to  address  a  particular 
bundle  of  architecture  concerns.  The  pattern  also  includes  technology  and  platform  decisions.  Of¬ 
ten,  little  additional  analysis  or  evaluation  is  performed,  since  the  quality  characteristics  of  the 
exemplar  are  already  established  through  its  existence  in  a  working  implementation. 
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4  Evaluation  of  Enterprise  Architectures  in  Practice 


This  section  discusses  how  enterprise  architectures  are  evaluated  in  practice. 

4.1  Enterprise  Architecture  Evaluation  Criteria 

The  scope  of  an  EA  evaluation  should  include  the  baseline  (“as-is”)  architecture,  the  target  (“to- 
be”)  architecture,  and  the  EA  roadmap.  The  evaluation  is  focused  on  evolution  from  baseline  to 
target  architectures. 

In  current  practice,  the  key  evaluation  criterion  is  line-of-sight1  from  roadmap  projects  and  the 
decisions  within  the  enterprise  and  solution  architectures  to  the  enterprise  business  goals  or  capa¬ 
bility  requirements.  The  business  architecture  provides  context  for  the  rest  of  the  EA  and  so  can 
provide  this  linkage  from  business  goals  to  architecture. 

4.2  Evaluation  Context 

Evaluation  occurs  at  multiple  levels:  entire  enterprise,  division  or  line-of-business,  and  project. 
Higher  levels  are  more  focused  on  alignment  than  technical  issues,  while  lower  levels  focus  on 
technical  concerns. 

Evaluations  should  consider  the 

•  current  (as-is)  state,  including  known  open  issues,  gaps  in  functionality  or  quality 
attributes,  and  inconsistencies  (in  solution  patterns,  for  example) 

•  proposed  changes  that  may  impact  the  architecture,  including  new  capabilities,  infrastruc¬ 
ture  changes,  and  operational  changes 

•  target  (to-be)  architecture,  including  roadmap  and  tradeoff  analysis  of  architecture  op¬ 
tions 

Current  evaluation  practice  gives  little  consideration  to  the  target  architecture.  It  may  happen  that 
the  EA  is  too  large  to  effectively  create  a  target  architecture;  in  such  cases  snapshots  of  sections  of 
the  architecture  may  suffice.  Several  participants  cited  examples  of  target  architectures  for  large 
enterprises,  so  size  is  not  a  hard  limiting  factor.  Like  the  target  architectures,  the  as-is  architecture 
may  also  be  incomplete,  which  hinders  effective  evaluation. 

In  discussions  of  current  EA  evaluation  practices,  there  was  a  clear  difference  between  govern¬ 
ment  and  industry  contexts.  Government  scale  is  often  larger,  and  government  organizations  seem 
to  have  better  discipline  around  practices  such  as  business  process  management.  On  the  industry 
side,  time-to-market  pressures  often  drive  decisions.  Budget  constraints  play  a  large  role  in  both 
contexts,  affecting  EA  development  and  evaluation. 

In  the  govermnent  context,  the  Clinger-Cohen  Act  and  associated  regulations  provide  minimum 
standards  for  EA  evaluation.  In  particular 

•  The  Government  Accountability  Office  (GAO)  audits  the  practices  used  to  create  the  EA. 


Participants  used  the  terms  line-of-sight  and  alignment  interchangeably. 
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The  Office  of  Management  and  Budget  (OMB  )  audits  how  the  organization  is  using  the 
EA  to  meet  business  performance  goals. 


The  SEI  principles  of  evaluating  an  architecture  to  determine  how  well  the  architecture  supports 
business  goals  seems  to  complement  the  OMB  standard. 

In  both  commercial  and  govermnent  contexts,  an  organization  generally  uses  the  same  governance 
process  for  all  projects,  with  no  tailoring  to  account  for  variation  in  project  scope  or  complexity. 
For  example,  changes  to  existing  capabilities  use  the  same  process  as  introduction  of  new  capabil¬ 
ities. 

One  particular  evaluation  context  is  what  one  participant  termed  “disruptive  requirements.”  These 
are  business  requirements  that  have  broad  architectural  significance  and  may  necessitate  changes 
in  common  infrastructure.  Such  requirements  should  be  escalated,  reviewed,  and  carefully  eva¬ 
luated. 

4.3  The  Role  of  Quality  Attributes  in  Enterprise  Architecture  Evaluation 

The  participants  had  varying  levels  of  familiarity  with  the  SEI  quality  attribute -based  approach  to 
architecture.  Among  participants  with  more  familiarity  there  was  consensus  that  current  EA  prac¬ 
tices  have  insufficient  focus  on  quality  attributes,  and  there  is  a  need  to  elevate  quality  attribute 
concerns  and  tradeoffs  to  first-class  status  in  EA  development  and  evaluation. 

There  are  some  standards  that  identify  EA  quality  attributes.  For  example,  Control  Objectives  for 
Information  and  related  Technology  (COBIT)  identifies 

•  effectiveness 

•  efficiency 

•  confidentiality 

•  integrity 

•  reliability 

•  availability 

•  compliance 

Other  typical  EA  quality  attributes  would  include 

•  profitability 

•  affordability 

•  scalability 

•  manageability 

•  aligmnent 

•  integration/interoperability 

•  sustainability 

•  agility 
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Presentations  of  EA  artifacts  by  several  participants  showed  that  quality  attributes  are  sometimes 
alluded  to  in  EA  documentation,  in  project  descriptions,  or  in  roadmap  annotations,  but  a  need 
exists  to  make  quality  attribute  concerns  more  explicit. 

Participants  discussed  the  use  of  end-to-end  business  threads  to  systematically  address  quality 
attribute  concerns.  One  participant  suggested  that  end-to-end  threads  could  be  used  to  identify  and 
resolve  contradictory  architecture  decisions.  These  threads  might  be  constructed  by  chaining  to¬ 
gether  existing  business  processes.  Discussion  ensued  about  the  cost  of  building  end-to-end 
threads,  and  agreement  that  more  study  in  this  area  is  necessary.  Pilot  applications  of  the  approach 
in  multiple  contexts  could  provide  insight. 

4.4  Evaluation  Methods 

In  current  practice,  EA  evaluation  is  not  performed  systematically  within  most  organizations. 
Technical  evaluations  are  more  often  performed  at  the  project/solution  level,  based  on  perceived 
importance  and  risk.  When  performed,  these  evaluations  use  ad  hoc  methods  relying  on  the  exper¬ 
tise  and  experience  of  the  reviewers.  (One  participant  characterized  the  approach  as  “heads  on 
sticks.”) 

The  participants  discussed  whether  business  threads  and  quality  attribute  scenarios  could  be  ap¬ 
plied  to  structure  and  systematize  EA  evaluations,  using  the  SoS  Architecture  Evaluation  Method, 
which  is  based  on  the  AT  AM  process  model  (See  Figure  2  in  Appendix  B).  Many  agreed  that  the 
approach  would  work  for  most  parts  of  an  EA  except  for  business  architecture  (for  example,  the 
TOGAF  Application,  Data,  and  Technology  Architectures).  The  return  on  investment  for  model¬ 
ing  and  analysis  in  the  business  architecture  domain  may  not  justify  performing  those  activities  to 
the  level  required  for  comprehensive  evaluation.  Work  in  this  area  is  typically  heavy  on  gover¬ 
nance  and  light  on  analysis.  Evaluations  focus  on  line-of-sight  to  business  goals  and  depend  on 
the  expertise  and  experience  of  the  reviewers.  Although  more  systematic  and  perhaps  quantitative 
evaluation  methods  might  be  helpful,  there  are  significant  technical,  cultural,  and  organizational 
challenges  to  adopting  them. 

Evaluations  using  the  AT  AM  process  model  could  be  performed  at  the  project/solutions  level, 
with  each  project  treated  as  a  system  of  systems,  and  risks  and  challenges  rolled  up  to  the  overall 
EA  level.  The  process  would  have  to  be  extended  to  include  evaluation  of  line-of-sight  to  business 
goals.  More  analysis,  including  pilot  engagements  in  several  contexts,  would  be  needed  to  vali¬ 
date  the  usefulness  of  AT  AM-based  methods. 

There  was  discussion  about  whether  evaluation  approaches  for  the  other  three  TOGAF  architec¬ 
tures  (Application  Architecture,  Data  Architecture,  and  Technology  Architecture)  would  be  the 
same,  or  if  each  type  of  architecture  would  require  different  methods.  In  particular,  participants 
identified  Data  Architecture  as  different — in  most  cases  involving  detailed  models  and  design 
standards,  and  evaluation  needs  to  encompass  the  data  models,  data  exchanges,  import/export,  and 
the  tools  to  manage  the  data.  In  general,  there  was  agreement  that  the  evaluation  process  might 
not  be  different  for  each  type  of  architecture,  but  the  quality  attributes  and  risk  impact  analysis 
would  certainly  be  different. 

Participants  described  varying  levels  of  investment  in  EA  evaluation.  One  organization  would 
spend  up  to  four  days  with  four  reviewers  to  evaluate  a  project  that  was  identified  as  architectural - 
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ly  significant,  but  the  same  organization  might  not  review  other  projects.  Another  organization 
spends  only  three  hours  with  eight  reviewers  for  most  projects.  Justifying  investment  in  architec¬ 
ture  evaluation  is  an  issue  in  many  organizations,  so  for  a  method  to  be  successful  in  practice,  it 
must  scale  down  to  short,  lighter  weight  evaluation  scenarios. 

4.5  Federation  and  Acquisition 

The  federation  of  enterprise  architectures  is  sometimes  required,  in  contexts  ranging  from  military 
coalitions  to  post-merger  corporate  integration.  Participants  identified  this  as  problematic  in  prac¬ 
tice  and  suggested  that  research  on  evaluation  of  federated  enterprise  architectures  might  produce 
valuable  results. 

There  was  discussion  about  the  role  of  EA  in  supporting  the  acquisition  of  services,  in  particular 
the  linkage  between  EA  and  contract  service-level  agreements.  Some  participants  were  trying  to 
implement  such  support,  but  open  questions  persisted  about  how  to  scale  it  to  the  enterprise  level 
and  how  to  include  acquired  services  in  the  evaluation  process. 


13  CMU/SEI-201 0-TN-023 


5  Summary 


5.1  Workshop  Findings 

At  the  close  of  the  workshop  the  participants  agreed  on  the  following  key  summary  points: 

1 .  Organizations  can  start  by  scoping  EA  as  IS/IT  (or  Application,  Data,  and  Technology  Archi¬ 
tectures,  using  TOGAF  nomenclature),  and  then  open  the  scope  to  include  Business  Architec¬ 
ture.  Although  it  may  not  be  necessary  to  consider  the  full  scope  of  the  business  architecture 
if  EA  is  limited  to  the  IS/IT  architecture,  supporting  enterprise-wide  strategic  decision  mak¬ 
ing  may  require  consideration  of  a  broader  business  architecture  scope. 

2.  Any  methods  for  EA  need  to  accommodate  high  variability  in  structure  (organization,  roles, 
relationships,  etc.)  between  the  IS/IT  and  the  business  architecture  owners.  This  variability 
stems  from  historical,  cultural,  and  maturity  differences  between  enterprises. 

Variability  includes  methods  and  evaluation  results  structured  for  various  process  frame¬ 
works,  including 

—  Capability  Maturity  Model  Integration  (CMMI®)  framework 

-  GAO  (defines  practices) 

-  OMB  EA  Assessment  Framework  (defines  EA  usage) 

-  National  Association  of  State  Chief  Infonnation  Officers  (NASCIO)  (originally  used 
GAO  and  OMB  frameworks,  then  developed  a  separate  self-assessment  framework) 

-  Gartner  Group 

-  TOGAF 

-  others 

3.  Line-of-sight,  or  alignment,  between  the  business  objectives  and  strategies  must  be  carried 
throughout  the  EA  (including  the  business  architecture).  This  alignment  must  be  developed 
down  to  an  individual  project  (with  documented  traceability)  and  evaluated  for  “goodness.” 

End-to-end  business  threads  augmented  with  quality  attribute  concerns  (as  described  in 
Quality  Attribute  Elicitation  for  Enterprise  Architectures  —  Business  Thread  Workshop  in 
Appendix  B)  seem  promising  for  capturing  alignment  from  architecture  to  business  goals. 
Threads  must  trace  back  to  business  goals  or  capability  objectives. 

4.  Use  of  end-to-end  business  threads  at  the  EA  level  may  be  beneficial  for  capturing  quality 
attribute  concerns  and  requirements  to  support  EA  design  and  evaluation.  Ideally,  these 
would  be  developed  incrementally  as  the  EA  evolves  and  then  maintained  as  part  of  the  ar¬ 
chitecture  knowledge  base.  Introducing  this  method  after  the  fact,  in  an  “up  and  running”  en¬ 
terprise  with  an  existing  EA,  may  be  prohibitively  time  consuming  and  expensive. 

A  pilot  study  in  this  area  may  help  clarify  the  effort  required  to  develop  end-to-end  busi¬ 
ness  threads. 

5.  It  appears  feasible  to  apply  quality  attribute  principles  to  drive  architecture  requirements  and 
evaluation  criteria  for  architecturally  significant  projects/solutions.  These  projects  would  be 
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treated  as  one  or  more  systems  of  systems.  Again,  a  pilot  study  to  use  the  Business  Thread 
Workshop  and  Enterprise  Architecture  Evaluation  (described  i nApplying  SoS  Approaches  to 
Enterprise  Architectures  in  Appendix  B)  would  quantify  the  cost  and  benefit  of  this  ap¬ 
proach. 

This  (or  another  systematic  approach  to  analysis  and  evaluation)  would  be  an  improve¬ 
ment  over  the  state  of  the  EA  evaluation  practice,  which  relies  on  expertise  and  expe¬ 
rience  to  assess  EA  designs  and  plans. 

A  need  also  exists  to  develop  decision  criteria  to  determine  when  to  use  quality  attribute- 
based  design  and  evaluation  methods  during  the  EA  life  cycle.  The  attention  paid  to  each 
project  is  different — for  example,  incremental  change  projects  tend  to  get  less  attention 
than  transformative  change  projects.  The  decision  criteria  must  accommodate  this. 
Checklists  or  tools  for  making  these  decisions  in  a  systematic  way  would  be  very  helpful. 

6.  Any  EA  analysis  and  evaluation  methods  should  avoid  drilling  down  to  software  architec¬ 
tures.  The  methods  should  include  software  and  system  architects  as  both  stakeholders  and 
design  collaborators,  but  EA  methods  should  only  look  at  attributes  of  the  software  and  sys¬ 
tem  architectures  that  are  significant  to  the  EA. 

5.2  Future  Work 

It  was  suggested  that  the  SEI  perform  pilot  studies  to  investigate  issues  that  may  arise  during 
overlay  of  the  EA  Engagement  Model  (shown  in  Figure  3  in  Appendix  B)  onto  various  EA  devel¬ 
opment  models,  such  as  the  TOGAF  Architecture  Development  Model  (ADM)  (The  Open  Group 
2009).  Issues  such  as  EA  evaluation  timing  and  scope,  structure  of  EA  evaluation  results,  and  do¬ 
cumentation  available  to  support  the  EA  evaluation  needs  to  be  aligned  between  the  EA  develop¬ 
ment  model  and  the  SEI  Engagement  Model. 

These  pilot  studies  should  also  examine  the  effort  required  to  introduce  and  apply  methods  such 
as  the  use  of  end-to-end  business  threads  for  capturing  quality  attribute  concerns  to  support  EA 
design  and  evaluation.  In  addition,  the  pilot  studies  should  investigate  how  service-level  contracts 
for  acquired  services  can  be  included  in  the  evaluation  scope. 

Methods  for  characterizing  the  scope  and  risk  of  EA  projects  are  needed.  This  characterization 
would  inform  investment  decisions  regarding  the  level  of  analysis  and  evaluation  that  is  appropri¬ 
ate  for  each  project. 

Development  of  a  comprehensive  mapping  between  the  EA  terminology  used  in  DoD,  other  gov¬ 
ernment  agencies,  and  commercial  practice  is  needed  to  allow  application  of  principles  and  prac¬ 
tices  across  domains. 
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Appendix  A  -  Survey  of  Enterprise  Architecture  Definitions 


•  (Ross,  Weill,  &  Robertson,  2006)  Enterprise  Architecture  is  the  organizing  logic  for 
business  processes  and  IT  infrastructure  reflecting  the  integration  and  standardization  re¬ 
quirements  of  the  firm’s  operating  model. .  .The  IT  unit  typically  addresses  four  levels  of 
architecture  below  the  enterprise  architecture:  business  process  architecture. .  .data  or  in¬ 
formation  architecture. .  .applications  architecture. .  .and  technology  architecture. .  .The 
tenn  enterprise  architecture  can  be  confusing  because  the  IT  unit  in  some  companies  re¬ 
fers  to  one  of  these  architectures — or  the  set  of  all  four  architectures — as  the  enterprise 
architecture. 

•  (Enterprise  Architecture  Research  Forum  2009)2  EA  is  the  continuous  practice  of  de¬ 
scribing  the  essential  elements  of  a  socio-technical  organization,  their  relationships  to 
each  other  and  to  the  environment,  in  order  to  understand  complexity  and  manage 
change. 

•  (The  Open  Group  2009)  There  are  four  architecture  domains  that  are  commonly  accepted 
as  subsets  of  an  overall  enterprise  architecture,3  all  of  which  TOGAF  is  designed  to  sup¬ 
port. 

1 .  The  business  architecture  defines  the  business  strategy,  governance,  organization, 
and  key  business  processes. 

2.  The  data  architecture  describes  the  structure  of  an  organization's  logical  and 
physical  data  assets  and  data  management  resources. 

3.  The  application  architecture  provides  a  blueprint  for  the  individual  application 
systems  to  be  deployed,  their  interactions,  and  their  relationships  to  the  core  business 
processes  of  the  organization. 

4.  The  technology  architecture  describes  the  logical  software  and  hardware  capabili¬ 
ties  that  are  required  to  support  the  deployment  of  business,  data,  and  application 
services.  This  includes  IT  infrastructure,  middleware,  networks,  communications, 
processing,  standards,  and  so  on. 

•  (The  Chief  Information  Officers  Council  1999,  in  the  Federal  Architecture  Framework 
Version  1.1)  Enterprise  Architecture  -  a  strategic  information  asset  base  which  defines 
the  mission,  the  information  necessary  to  perform  the  mission,  the  technologies  necessary 
to  perfonn  the  mission,  and  the  transitional  processes  for  implementing  new  technologies 
in  response  to  the  changing  mission  needs.  An  enterprise  architecture  includes  a  baseline 
architecture,  target  architecture,  and  a  sequencing  plan. 

•  (National  Institutes  of  Health  2008)  Enterprise  architecture  is  a  comprehensive  frame¬ 
work  used  to  manage  and  align  an  organization's  Information  Technology  (IT)  assets, 
people,  operations,  and  projects  with  its  operational  characteristics.  In  other  words,  the 


A  recent  survey  by  The  Open  Group  selected  this  definition  as  the  most  popular  (Booch,  2010). 
Interestingly,  TOGAF  never  explicitly  defines  the  term  enterprise  architecture. 
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enterprise  architecture  defines  how  infonnation  and  technology  will  support  the  business 
operations  and  provide  benefit  for  the  business. 

It  illustrates  the  organization’s  core  mission,  each  component  critical  to  performing  that 
mission,  and  how  each  of  these  components  is  interrelated.  These  components  include 

-  Guiding  principles 

-  Organization  structure 

-  Business  processes 

—  People  or  stakeholders 

-  Applications,  data,  and  infrastructure 

-  Technologies  upon  which  networks,  applications  and  systems  are  built 
Guiding  principles,  organization  structure,  business  processes,  and  people  don’t 
sound  very  technical.  That’s  because  enterprise  architecture  is  about  more  than  technolo¬ 
gy.  It  is  about  the  entire  organization  (or  enterprise)  and  identifying  all  of  the  bits  and 
pieces  that  make  the  organization  work. 

•  (Lapkin,  2006,  for  the  Gartner  Group  )  Enterprise  architecture  is  the  process  of  translating 
business  vision  and  strategy  into  effective  enterprise  change  by  creating,  communicating 
and  improving  the  key  principles  and  models  that  describe  the  enterprise’s  future  state 
and  enable  its  evolution.  The  scope  of  the  enterprise  architecture  includes  the  people, 
processes,  information  and  technology  of  the  enterprise,  and  their  relationships  to  one 
another  and  to  the  external  enviromnent.  Enterprise  architects  compose  holistic  solutions 
that  address  the  business  challenges  of  the  enterprise  and  support  the  governance  needed 
to  implement  them. 

•  (SearchCIO.com  2007)  An  enterprise  architecture  (EA)  is  a  conceptual  blueprint  that  de¬ 
fines  the  structure  and  operation  of  an  organization.  The  intent  of  an  enterprise  architec¬ 
ture  is  to  determine  how  an  organization  can  most  effectively  achieve  its  current  and  fu¬ 
ture  objectives  (also  used  by  [Platt  2002]  at  Microsoft  MSDN  and  [Ruest  2006]  at  IBM 
DeveloperWorks). 

•  (Zachman  2008)  Architecture  is  the  set  of  descriptive  representations  that  are  required  to 
create  an  object. 
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Appendix  B  -  SEI  Enterprise  Architecture  Analysis  and 
Evaluation  Engagement  Model 


On  Day  1  of  the  workshop,  John  Klein  from  the  SEI  made  a  presentation  that  the  SEI’s  current 
work  in  SoS  architecture  analysis  and  evaluation,  and  proposed  extensions  to  those  methods  for 
EA  analysis  and  evaluation.  These  methods  were  discussed  frequently  during  the  workshop  and 
are  referenced  in  the  Workshop  Findings  and  Future  Work  in  Section  5  above,  and  so  the  material 
from  the  presentation  is  included  here. 

Introduction 

We  believe  that  EA  is  critical  to  achieving  business  goals  and  that  architectures  are  shaped  by 
quality  attribute  requirements  (such  as  those  identified  in  Section  4.3  above).  So  we  consider  the 
following  questions: 

•  Flow  do  we  ensure  that  we  have  correctly  and  completely  translated  business  goals  into 
quality  attribute  requirements? 

•  How  do  we  ensure  that  these  quality  attribute  requirements  are  reflected  in  the  tradeoffs 
and  decisions  that  shaped  the  EA? 

We  begin  by  reviewing  the  SEI  perspective  on  architecture-centric  engineering.  Next  we  discuss 
how  that  approach  scales  from  its  original  software  context  through  systems  and  systems  of  sys¬ 
tems.  We  review  the  SEI  methods  applicable  to  systems  and  systems  of  systems,  and  finally  pro¬ 
pose  how  those  methods  can  be  extended  to  apply  to  enterprise  architectures. 

An  Architecture-Centric  Perspective 

The  SEI  approach  to  architecture  is  grounded  in  the  following  tenets: 

•  Every  system  has  an  architecture,  regardless  of  scale. 

•  Architecture  is  the  appropriate  abstraction  for  reasoning  about  business  or  mission  goal 
satisfaction. 

•  Quality  attributes  have  a  dominant  influence  on  a  system’s  architecture. 

•  Value  derived  from  business  and  mission  goals  governs  quality  attribute  tradeoffs. 

•  Well-founded,  cost-effective  measurements  and  analyses  are  the  bases  for  acquiring  con¬ 
fidence  about  system  properties. 

•  Architectural  prescriptions  must  be  demonstrably  satisfied  by  the  implementation. 

•  Architectural  decisions  made  today  must  appropriately  reflect  the  drivers  of  system 
change. 

We  define  the  architecture  of  a  computing  system  as  the  structures  of  the  system,  which  comprise 
software  elements,  the  externally  visible  properties  of  those  elements,  and  the  relationships  be¬ 
tween  them  (Bass  2003).  This  definition  is  equally  applicable  to  cases  where  the  architecture  is 
accidental  and  to  cases  where  the  architecture  is  intentional. 
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In  an  intentional  architecture,  the  structures  result  from  decisions  made  by  an  architect.  Each  deci¬ 
sion  is  a  tradeoff  which  promotes  some  qualities  of  the  system  while  diminishing  other  qualities. 
The  traceability  of  quality  attributes  to  business  or  mission  goals  provides  the  decision  criteria  for 
these  tradeoffs.  In  the  case  of  accidental  architectures,  decisions  may  be  made  by  any  stakeholder, 
and  tradeoffs  are  not  systematically  traceable  to  business  goals. 

Scaling  to  Address  Enterprise  Architecture 

The  SEI  has  extended  methods  for  architecture  analysis  and  evaluation  in  a  relatively  direct  man¬ 
ner  from  software  up  through  SoS  (Gagliardi  2009).  All  of  these  methods  are  characterized  by 

•  direct  stakeholder  participation  in  specification  of  quality  attribute  requirements  and  in 
architecture  evaluation 

•  use  of  concrete  scenarios  or  end-to-end  threads  to  define  quality  attribute  requirements 
and  as  the  basis  for  architecture  evaluation 

•  recognition  that  none  of  the  methods  are  exhaustive — the  results  depend  on  engaging  suf¬ 
ficient  diversity  of  stakeholders  to  address  the  most  important  quality  attribute  require¬ 
ments. 

In  considering  enterprise  architectures,  a  way  to  extend  these  methods  was  not  obvious.  Part  of 
the  difficulty  may  be  due  to  definitional  mismatch.  Appendix  A  lists  a  number  of  definitions  of 
EA,  which  share  the  following  themes: 

•  An  EA  is  composed  of  (or  realized  by)  four  “sub-architectures” — 

(1)  business  architecture,  (2)  information  or  data  architecture,  (3)  application  architecture, 
and  (4)  technology  or  infrastructure  architecture. 

•  EA  refers  to  both  a  process  and  the  artifacts  produced  by  the  process. 

•  The  elements  of  an  EA  include  people. 

The  notions  of  structure,  quality  attributes,  and  tradeoffs  are  not  explicit  in  most  of  the  discussions 
of  EA.  (This  and  other  differences  between  the  genres  of  software,  system,  system  of  systems,  and 
EA  were  explored  in  detail  at  an  earlier  SEI  workshop  [Bergey  2009]).  Furthermore,  the  diversity 
of  stakeholders  and  the  number  of  scenarios  or  business  processes  to  consider  in  an  SoS  architec¬ 
ture  or  EA  can  become  intractable,  risking  spotty  coverage  of  quality  attribute  requirements  or 
leading  to  a  very  long  process  to  achieve  adequate  breadth. 

These  differences  could  lead  one  to  conclude  that  enterprise  architectures  are  fundamentally  dif¬ 
ferent  from  system  architectures  (Booch  2010).  On  the  other  hand,  John  Zachman,  the  father  of 
EA,  asserted  that  “Architecture  is  Architecture  is  Architecture”  (Zachman  2007).  This  implies  that 
we  should  be  able  to  apply  the  principles  and  practices  that  have  proven  effective  for  analyzing 
and  evaluating  software  architectures  to  architectures  for  systems,  systems  of  systems,  and  enter¬ 
prises. 

Architecture  evaluations  based  on  methods  such  as  the  AT  AM  (Clements  2002)  or  the  SEI  Sys¬ 
tem  of  Systems  Architecture  Evaluation  Method  (Gagliardi  2009)  begin  by  identifying  business  or 
mission  goals  for  the  system.  The  business  goals  are  then  reflected  in  quality  attribute  require¬ 
ments  and  specified  using  concrete  scenarios.  The  scenarios  are  used  to  analyze  the  architecture  to 
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identify  decisions  and  tradeoffs  and  then  to  determine  if  the  decisions  and  tradeoffs  reflected  in 
the  architecture’s  structures  are  consistent  with  the  quality  attribute  requirements. 

In  attempting  to  extend  existing  methods,  we  began  by  questioning  the  primacy  of  quality 
attribute  requirements  as  the  driver  for  EA.  In  looking  at  the  definitions  of  EA,  we  considered 
other  perspectives  on  EA  that  lead  to  different  evaluation  approaches: 

•  EA  is  a  process.  We  can  evaluate  the  quality  of  the  process  and  adherence  to  the  process 
with  methods  like  CMMI. 

•  The  EA  process  is  carried  out  by  individuals  and  teams  working  within  an  organization. 
We  can  evaluate  the  capability  of  the  people,  teams,  and  organization  using  a  method  like 
the  SEI  Architecture  Capability  Assessment  (Bass  2009). 

•  Business  processes  are  a  first-order  element  in  EA.  We  could  use  an  Organizational 
Coordination  Theory  perspective  to  evaluate  alignment  between  the  business  processes 
and  organizational  structures  (Bass  2008). 

While  these  approaches  may  complement  an  architecture-centric  evaluation  approach,  we  take  the 
position  that  methods  building  on  the  AT  AM  and  the  SEI  System  of  Systems  Architecture  Evalu¬ 
ation  Method  are  necessary  to  adequately  evaluate  an  EA  to  ensure  alignment  of  the  EA  to  busi¬ 
ness  goals. 

Background  -  SEI  Methods  for  Software-Reliant  Systems  and  Systems  of  Systems 

The  following  sections  provide  background  on  mature  SEI  methods  for  analyzing  and  evaluating 
software-reliant  systems  and  systems  of  systems.  These  methods  provide  the  basis  of  the  EA  me¬ 
thods  described  below. 

Quality  Attribute  Elicitation  for  Software-Reliant  Systems  and  Systems  of 
Systems 

Organizations  frequently  have  difficulty  developing  quality  attribute  requirements  (Barbacci 
2003).  The  SEI  Quality  Attribute  Workshop  was  developed  to  provide  a  systematic  method  for 
quality  attribute  requirement  elicitation  (Barbacci  2003). 

The  method  brings  together  as  many  as  20  stakeholders,  and  uses  scenarios  to  help  them  express 
their  quality  attribute  requirements  for  the  system.  It  has  been  used  to  elicit  quality  attribute  re¬ 
quirements  for  dozens  of  software-reliant  systems. 

The  Quality  Attribute  Workshop  (QAW)  is  a  facilitated  method  that  engages  system  stakeholders 
early  in  the  system  development  life  cycle  to  discover  the  driving  quality  attributes  of  a  software¬ 
intensive  system.  The  QAW  is  system-centric  and  stakeholder  focused;  it  is  used  before  the  soft¬ 
ware  architecture  has  been  created.  The  QAW  provides  an  opportunity  to  gather  stakeholders  to¬ 
gether  to  provide  input  about  their  needs  and  expectations  with  respect  to  key  quality  attributes 
that  are  of  particular  concern  to  them. 

As  originally  developed,  the  method  takes  a  “bottom-up”  brainstorming  approach,  and  the  output 
is  simply  a  list  of  quality  attributes,  concerns,  and  scenarios.  In  practice,  sometimes  the  interme¬ 
diate  results  are  structured  into  a  utility  tree  partway  through  the  workshop,  and  the  elicitation 
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then  continues  in  a  more  “top-down”  fashion.  In  the  case  of  very  large  or  complex  systems,  there 
may  be  a  series  of  QAWs,  each  focused  on  a  different  slice  of  functionality. 

The  use  of  scenarios  to  help  stakeholders  express  quality  attribute  requirements  extends  through 
many  of  the  SEI  architecture-centric  methods.  A  basic  scenario  describes  how  the  system  re¬ 
sponds  to  a  particular  stimulus  under  a  particular  operating  mode  or  environment.  The  basic  sce¬ 
nario  can  be  extended  to  include  the  stimulus  source,  specific  response  measure,  and  system  ele¬ 
ments  involved  in  the  scenario.  This  scenario-based  methodology  has  proven  successful  in 
eliciting  actionable  quality  attribute  requirements,  so  that  architects  do  not  have  to  rely  on  vague 
stakeholder  requests  like  “highly  available,”  “low  latency,”  and  “user  friendly.” 

The  QAW  has  been  extended  to  address  the  needs  of  military  systems  of  systems.  The  Mission 
Thread4  Workshop  is  also  a  facilitated  process  that  brings  together  SoS  stakeholders  to  both  aug¬ 
ment  existing  mission  threads  with  quality  attribute  considerations  that  will  shape  the  SoS  archi¬ 
tecture  and  identify  SoS  architectural  challenges. 

Architecture  Evaluation  of  Software-Reliant  Systems  and  Systems  of  Systems 

The  SEI  Architecture  Tradeoff  Analysis  Method  (ATAM)  was  originally  developed  for  evaluating 
the  software  architecture  of  a  software-reliant  system.  It  brings  together  a  trained  evaluation 
team,  the  decision  makers  for  the  system  and  architecture,  and  representatives  of  the  architecture’s 
stakeholders.  The  facilitated  process  helps  stakeholders  to  ask  the  right  questions  to  discover  po¬ 
tentially  problematic  architectural  decisions. 

The  ATAM  is  conceptually  depicted  in  Figure  2.  The  method  begins  by  identifying  business  goals 
for  the  system.  The  business  goals  are  then  reflected  in  quality  attribute  requirements  and  speci¬ 
fied  using  concrete  scenarios.  The  scenarios  are  used  to  analyze  the  architecture  and  determine  if 
the  decisions  and  tradeoffs  that  led  to  the  architecture’s  structures  are  consistent  with  the  quality 
attribute  requirements.  The  method  identifies  risks  (potentially  problematic  decisions)  and  non¬ 
risks,  and  explicitly  identifies  tradeoffs  between  quality  attributes  and  sensitivity  points  (decisions 
that  significantly  affect  the  ability  of  the  architecture  to  achieve  a  particular  quality  attribute  re¬ 
sponse).  Risks  are  summarized  into  “risk  themes”  that  provide  an  executive  summary  of  the  eval¬ 
uation  and  help  organize  risk  mitigation  planning. 

It  is  up  to  the  organization  developing  the  architecture  and  system  to  decide  whether  and  how  to 
address  the  risks.  The  quantification  of  each  risk  and  associated  mitigation  costs  can  only  be  de¬ 
termined  within  the  business  and  organizational  context  where  the  system  is  being  developed.  The 
SEI  Cost  Benefit  Analysis  Method  (CBAM)  provides  a  structured  method  for  analyzing  alterna¬ 
tive  courses  of  action  (Clements  2002). 

The  ATAM  was  recently  extended  to  evaluate  software  and  systems,  that  is,  the  software  and 
associated  electrical,  mechanical,  and  other  physical  elements  of  the  software-reliant  system 
(Gagliardi  2009).  For  these  evaluations,  domain  experts  from  the  related  physical  disciplines  are 
given  just-in-time  training  to  qualify  to  be  evaluation  team  members,  and  the  quality  attribute 
scope  is  extended  to  the  physical  domains  of  interest. 


A  mission  thread  is  a  sequence  of  activities  and  events  beginning  with  an  opportunity  to  detect  a  threat  or  ele¬ 
ment  that  ought  to  be  attacked  and  ending  with  a  commander’s  assessment  of  damage  after  an  attack. 
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Figure  2:  Architecture  Analysis  Tradeoff  Method  (AT AM)  Process  Flow 

The  principles  underlying  the  AT  AM  have  been  further  extended  to  create  the  SEI  System  of  Sys¬ 
tems  Architecture  Evaluation  method,  which  has  been  applied  in  C4ISR  contexts  (Gagliardi 
2009).  The  method  uses  mission  threads  augmented  with  quality  attribute  concerns  (generated 
during  Mission  Thread  Workshops  described  above)  instead  of  scenarios  to  express  the  quality 
attribute  requirements  for  the  SoS.  Mission  threads  are  selected  for  analysis  to  reflect  concerns  in 
several  broad  categories:  operational  concerns  (tactical  operation  of  the  SoS),  sustainment  con¬ 
cerns  (field  maintenance,  updates,  and  training),  and  development  (including  test,  integration,  and 
associated  processes  and  facilities).  Elowever,  the  complexity  of  SoS  architectures,  along  with  the 
number  and  breadth  of  stakeholders,  is  not  conducive  to  performing  exhaustive  analysis,  and  so 
the  success  of  the  method  is  sensitive  to  which  mission  threads  are  selected  for  analysis.  It  is  for 
this  reason  that  the  method  is  targeted  to  “first  pass”  risk  identification. 

Applying  SoS  Approaches  to  Enterprise  Architectures 

The  SEI  approach  to  analyzing  and  evaluating  enterprise  architectures  is  based  on  the  methods 
used  for  system-of-systems  architectures.  Figure  3  below  shows  how  the  SEI  SoS  Engagement 
Model  might  be  extended  to  apply  to  enterprise  architectures.  In  particular,  the  Mission  Thread 
Workshop  is  modified  as  the  Business  Thread  Workshop,  and  the  SoS  Architecture  Evaluation  is 
extended  as  the  Enterprise  Architecture  Evaluation.  In  each  case,  the  extensions  are  straightfor¬ 
ward,  and  are  described  below. 

Quality  Attribute  Elicitation  for  Enterprise  Architectures  -  Business  Thread 
Workshop 

The  Business  Thread  Workshop  (BTW)  is  a  facilitated  engagement  where  the  EA  stakeholders 
augment  a  business  thread  with  quality  attribute  considerations.  A  business  thread  is  defined  as  an 
end-to-end  flow  through  the  enterprise,  perhaps  encompassing  multiple  business  processes.  An 
example  might  be  customer  order  placement  through  a  contact  center  through  order  fulfillment 
through  billing  and  accounts  receivable  through  delivery  through  return  authorization  through 
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receiving  and  accounts  payable.  At  each  step  in  the  flow,  quality  attribute  considerations 
(throughput,  latency,  measurability,  auditability,  etc.)  are  attached  to  the  step.  Additionally,  over¬ 
arching  quality  attribute  concerns  are  identified. 

Based  on  our  experience  with  Mission  Thread  Workshops  and  SoS  Architecture  Evaluations, 
there  are  three  broad  categories  of  business  threads  that  may  be  considered  during  the  BTW: 

•  Core  Business  -  these  threads  trace  through  the  core  business  processes  of  the  enterprise. 
The  thread  described  above  is  an  example  of  a  Core  Business  thread. 

•  Operations  Threads  -  these  threads  trace  through  support  operations  processes.  Examples 
include  deployment,  migration,  day-to-day  management,  training,  and  disaster  recovery. 

•  Development  Threads  -  including  development,  test,  and  integration. 

Additionally,  business  threads  can  be  categorized  as 


•  “As-ls”  —  these  threads  reflect  as-is  capabilities  that  must  be  maintained  as  the  EA 
changes,  for  example  during  integration  of  an  acquired  company. 

•  “To-Be”  —  these  threads  reflect  well-defined  future  capabilities  that  must  be  supported  in 
a  new  or  evolved  architecture. 

•  “What-If  ’  -  these  threads  are  analogous  to  “Growth  Scenarios”  in  the  AT  AM,  exploring 
opportunities  and  testing  the  limits  of  the  EA. 


The  SE1  has  piloted  a  series  of  BTWs  with  a  financial  services  customer  to  develop  analysis  and 
evaluation  scenarios,  with  generally  positive  results. 


Business  Drivers 

V 

y 

Enterprise  Architecture  Risks 

Problematic  systems  with 
augmented  business  threads 


Augmented  Business  Threads 
Enterprise  Arch.  Challenges 


Enterprise  Architecture 
System  Architectures 


System  and  Software 
Architecture 


System  and  Software 
Architecture  Risks 


Enterprise  Architecture  Development 
System  and  Software  Architectures  Development,  Risk  Management 


Figure  3:  SEI  Enterprise  Architecture  Engagement  Model 
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Architecture  Evaluation  for  Enterprise  Architectures 

The  SEI  Enterprise  Architecture  Evaluation  method  has  been  developed  to  identify  EA  risks.  It  is 
identical  to  the  SoS  Architecture  Evaluation  method,  except  that  is  uses  augmented  business  threads 
instead  of  augmented  mission  threads.  Like  the  SoS  Architecture  Evaluation  Method,  it  is  sensitive 
to  the  threads  chosen  for  analysis  and  to  stakeholder  participation.  It  is  concerned  with  stakeholder 
participation  in  both  (1)  the  BTWs  that  augment  the  business  threads  with  quality  attributes  concerns 
and  (2)  the  architecture  evaluation  itself. 

The  evaluation  is  carried  out  by  walking  each  augmented  business  thread  through  the  architecture, 
and  having  the  architect  use  EA  documentation  artifacts  to  demonstrate  how  the  architecture  will 
support  the  functionality  and  quality  attribute  requirements  embodied  in  the  augmented  thread.  In 
cases  where  risks  indicate  that  the  underlying  systems  may  not  adequately  satisfy  the  architecture 
requirements,  then  more  detailed  evaluation  of  those  systems  using  the  System  and  Software 
ATAM  may  be  warranted. 

Since  this  workshop  was  held,  the  SEI  recently  performed  an  Enterprise  Architecture  Evaluation 
to  evaluate  the  “Enterprise  Services  and  Processes”  part  of  an  EA.  This  included  the  generation  of 
end-to-end  business  threads,  augmented  with  quality  attribute  considerations  from  various  stake¬ 
holders.  The  end-to-end  thread  generation  and  augmentation  was  simple  and  straightforward. 
Seven  end-to-end  threads  were  developed  and  augmented  in  less  than  one  half-day.  Stakeholders 
participating  in  the  evaluation  felt  that  these  end-to-end  threads  provided  adequate  coverage  for 
the  entire  EA.  Included  within  the  scope  of  this  evaluation  were  the  (1)  business  processes,  (2) 
enterprise  services,  (3)  user  interactions,  (4)  engineering  change  processes,  (5)  engineering  devel¬ 
opment,  (6)  integration,  and  (7)  deployment  processes.  The  evaluation  was  executed  in  one  day 
and  successfully  identified  risks,  issues,  and  non-risks  for  the  Enterprise  Services  and  Processes 
for  the  EA.  The  EA  evaluation  method  appears  to  be  very  promising  for  use  in  evaluating  busi¬ 
ness,  data,  application,  and  technology  architecture  domains. 
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Acronyms  and  Abbreviations 


The  following  alphabetical  list  contains  the  acronyms,  abbreviations,  and  their  meanings  as  used 
in  this  report. 


ATAM 

Architecture  Tradeoff  Analysis  Method 

BTW 

Business  Thread  Workshop 

C4ISR 

Command,  Control,  Communications,  and  Computing  for  Intelligence,  Surveillance,  and  Re¬ 
connaissance 

CIO 

Chief  Information  Officer 

COBIT 

Control  Objectives  for  Information  and  related  Technology 

DoD 

Department  of  Defense 

DoDAF 

DoD  Architecture  Framework 

EA 

Enterprise  Architecture 

GAO 

Government  Accountability  Office 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IS 

Information  Systems 

IT 

Information  Technology 

MTW 

Mission  Thread  Workshop 

OMB 

Office  of  Management  and  Budget 

QAW 

Quality  Attribute  Workshop 

ROI 

Return  on  Investment 

SoS 

System  of  Systems 

TOGAF 

The  Open  Group  Architecture  Framework 

UML 

Unified  Modeling  Language 
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